Image Caption Generation with Recursive Neural Networks

The ability to recognize image features and generate accurate, syntactically reasonable text descriptions is important for many tasks in computer vision. Auto-captioning could, for example, be used to provide descriptions of website content, or to generate frame-by-frame descriptions of video for the vision-impaired. In this project, a multimodal architecture for generating image captions is explored. The architecture combines image feature information from a convolutional neural network with a recurrent neural network language model, in order to produce sentence-length descriptions of images. An attention mechanism is used, which allows the network to focus on the most relevant image features at each time-step during caption generation.